What changes were proposed in this pull request?

JIRA: https://issues.apache.org/jira/browse/SPARK-43427

Protobuf supports unsigned integer types, including uint32 and uint64. When deserializing protobuf values with fields of these types, from_protobuf currently transforms them to the spark types of:

uint32 => IntegerType
uint64 => LongType

IntegerType and LongType are signed integer types, so this can lead to confusing results. Namely, if a uint32 value in a stored proto is above 2^31 or a uint64 value is above 2^63, their representation in binary will contain a 1 in the highest bit, which when interpreted as a signed integer will be negative (I.e. overflow). No information is lost, as IntegerType and LongType contain 32 and 64 bits respectively, however their representation can be confusing.

In this PR, we add an option (upcast.unsigned.ints) to allow upcasting unsigned integer types into a larger integer type that can represent them natively, i.e.

uint32 => LongType
uint64 => Decimal(20, 0)

I added an option so that it doesn't break any existing clients.

Example of current behavior

Consider a protobuf message like:

syntax = "proto3";

message Test {
  uint64 val = 1;
}

If we compile the above and then generate a message with a value for val above 2^63:

import test_pb2

s = test_pb2.Test()
s.val = 9223372036854775809 # 2**63 + 1
serialized = s.SerializeToString()
print(serialized)

This generates the binary representation:

b'\x08\x81\x80\x80\x80\x80\x80\x80\x80\x80\x01'

Then, deserializing this using from_protobuf, we can see that it is represented as a negative number. I did this in a notebook so its easier to see, but could reproduce in a scala test as well:

Precedent
I believe that unsigned integer types in parquet are deserialized in a similar manner, i.e. put into a larger type so that the unsigned representation natively fits. https://issues.apache.org/jira/browse/SPARK-34817 and #31921. So an option to get similar behavior would be useful.

Why are the changes needed?

Improve unsigned integer deserialization behavior.

Does this PR introduce any user-facing change?

Yes, adds a new option.

How was this patch tested?

Unit Testing

Was this patch authored or co-authored using generative AI tooling?

No